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SUMMARY OF THE REPORT 



During summer 1986, the Division of Special Education 
(D.S.E.) of the New York City Board of Education operated a 
Chapter 1 Summer Program, Reading and Math with Athletics, which 
served a total of 1,625 students at 13 sites for six weeks. In 
this program, students received morning instruction: one period 
per day each in reading, mathematics, and in athletic and 
recreational activities. The Big Apple Sports Program sponsored 
the athletic and recreational activities for a four-week period. 
In addition, students had one cultural or recreational trip per 
week. Program personnel incorporated athletic activities into 
reading and math instruction. They used a holistic, meaning- 
centered approach to reading instruction and emphasized 
integrating students* personal experiences wi^h language arts. 
In math, instruction emphasized word problems and practical 
skills. To measure academic progress, students took pretests and 
posttests with the Metropolitan Achievement Tests in Reading and 
Hath. 

The Office of Educat onal Assessment (O.E.A.) evaluated 
program implementation and student outcomes. To assess 
implementation, O.E.A. field consultants visited seven program 
sites to observe instruction, interview staff, and examine 
records. Results indicated that the program was satisfactorily 
implemented at all sites. 

Program staff successfully incorporated sports themes into 
academic instruction; they indicated that the sports curriculum 
component and the trips enhanced student motivation to learn 
reading and math. Staff at several sites reported that the Big 
Apple Program was successful and recommended that it be extended 
for the full six weeks of che program. Staff at some sites 
indicated that they wanted more audio-visual equipment, math 
manipulative materials, and consumable instructional materials 
for students to take home. 

At most sites program staff reported that test materials had 
arrived on time. Teachers liked the Special Education Teaci^^r's 
Guide in Commu nication Arts but would have preferred the 
inclusion of complete math lessons and lessons for lower- 
functioning readers. They requested that teacher trainers 
provide more training in the holistic approach to reading and 
schedule periodic on-site meetings. They further requested that 
a duplicating machine and telephone be available at every site. 

Classroom teachers selected the pretest level according to 
pre-program reading and math scores supplied for each student. 
These scores were not available, however, for all students. 

The evaluation objectives for the program were that 75 
percent of the students would show a gain of two raw score points 
from the pretest to the posttest in reading and math. The 
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I . INTRODUCTION 

This report presents the Office of Educational Assessment's 
(O.E.A.'s) evaluation of the 1986 E.C.I. A. Chapter 1 Summer 
Program, Reading and. Math with Athletics. The Division of 
Special Education (D.S.E.) of the New York City Board of 
Education operated the program, which provided remedial reading 
and mathematics instruction to 1,625 eligible students with mild 
to moderate handicapping conditions. The Big Apple Sports 
Programs and other services provided supplementary athletic, 
recreational, and cultural activities. The program integrated 
these non-academic activities into the reading and math 
instruction. 

The program, which operated five days a week from 8:30 A.M. 
to 12:30 P.M. for six weeks beginning July 7, 1986, was located 
at 13 sites in the five special education regions. Students 
received an average of one hour of reading instruction, one hour 
of math instruction, and participated in one hour of sports or 
other recreational activities daily. Breakfast and lunch took a 
half hour each. They took approximately one off-site trip every 
week . 

Staff used student pretest scores on the Metropolitan 
Achievement Test (M.A.T.) in reading and math as a basis for the 
development of individualized instructional plans. To assess 
academic gains, they posttested students during the last week of 
the program with the same instrument. 

The 1985 B.C. I. A. Chapter 1 Summer Program, Reading and Math 
with Athletics served approximately 1,900 students and used the 
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M.A.T. for pre- and posttesting. Over 75 percent of the students 
in both reading and math showed a pre test/posttest raw score 
gain. Recommendations for that program cycle included extending 
the sports program over the six weeks of the program, providing 
more audio-visual equipment and math manipulative, providing an 
adequate supply of test materials in a timely fashion, obtaining 
pre-program reading and math scores for all incoming students,, 
and expanding the Te acher's Guide to better cover the needs of 
lower-functioning students and to include math lessons. 

STAFF 

The staff of the Chapter I Summer Program consisted of 148 
classroom teachers who provided remedial reading and math 
instruction. The same number of paraprof essionals assisted them. 
Thirteen unit teacherscoord^^iated the activities of all classes 
and 13 site supervisors supervised the individual sites. The 
unit teachers were responsible for registration, testing, 
placement, transportation, and scheduling. The site supervisor 
took responsibility for coordination and supervision at each 
site. Five regional coordinators had responsibilities for 
program start-ups and overall coordination. In addition, there 
was at least one Big Apple Sports staff jnember at each site. 
Factors that influenced site selection included a central 
location for bus routes, the availability of adequate classroom 
and gymnasium space for a large number of classes, a breakfast 
and lunch program in pla^e, and being open for other summer 
programs. 
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REPORT FORMAT 

This report is organized as follows: Chapter II describes 
the evaluation methodology; Chapter III presents an analysis of. 
the qualitative and quantitative findings of the evaluation by 
describing program activities, examining student outcomes, and 
identifying factors which facilitated or hindered program 
success; and Chapter IV presents conclusions and recomrendation 
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II. EVALUATION METHODOLOGY 



EVALUATION QUESTIONS 

The evaluation assessed two major areas: program 
implementation and ouccomes. Evaluation questions included the 
following: 

Process/Iinpleinentation 

^ What was the level and quality of program 
implementation? 

^ Did teachers use a holistic approach to reading? 

<^ Did program staff integrate the athletic and cultural 
activities with the reading and math instruction? 

Outcome 

^ What was the program's impact on student achievement? 

EVALUATION OBJECTIVES 

O.E.A. addressed the following evaluation objectives. By 

August 15, 1986, 75 percent of participating students will have: 

^ shown a raw score gain from the pretest to the posttest 
of at least two points on the Reading M.A.T.; 

shown a raw score gain from the pretest to the posttest 
of at least two points on the Math M.A.T. 

SAMPLE 

Field consulranrs visited seven of the 13 program sites to 
observe instruction and sports activities, inspect records, and 
interview staff rerbers. They observed seven reading classes, 
seven math classes, and seven athletic periods; they interviewed 
14 classroom teachers mine percent of the total) , seven site 
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supervisors (54 percent of the total), and four unit teachers (31 
percent of the total). 

Teachers submitted achievement test data on all 1,625 
Chapter 1- eligible student!^. Since gains could only be 
calculated for those students for whom there were pre- and 
posttest scores and because it was determined that only the 
scores of students who attended a minimum of 20 days should 
enter into the analysis, the sample for purposes of analyzing 
objective attainment consisted of 1,374 students for reading and 
1,332 students for math. 

INSTRUMENTS 

O.E.A. developed two observation guides for field 
consultants. One enabled them to document the adequacy of 
physical facilities, records, instruction, and materials. They 
used the other guide to document the type of sports activities 
observed, along with the extent of student and staff 
participation. O.E.A. developed interview guides for their 
interviews of program personnel in order to obtain information on 
instructional activities, testing, materials, and staff training. 
Teachers used an 0. E .A. -developed data retrieval form to 
contribute information about the target population and enter data 
on attendance as well as on pre- and posttest scores. 

ANALYSIS 

O.E.A. analyzed the achievement of program participants on a 
pretest/posttest basis. Test scores of only those students with 
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both pre- and posttest scores were included, permitting O.E.A. to 
examine the achievement data of fewer students than were in the 
program. Additionally, O.E.A. examined the achievement data only 
of those students attending a minimum of 20 sessions. Those 
students remaining in the sample undoubtedly accomplished more 
objectives than students in the program a shorter time or for 
fewer sessions. In both reading and mathematics, O.E.A. 
calculated the frequency distribution of raw score gains and 
computed the number and percent of students showing a raw score 
gain by test level. As a norm referenced test, the interval 
between the pre- and posttesting was too short to allow a valid 
comparison. Accordingly, assessment employed raw rather than 
standard scores. Additionally, O.E.A. computed statistical 
significance and effect size of mean raw score gains by test 
level for all pairs of matched scores. 

Interviews of Chapter 1 teachers, special education 
classroom teachers, site supervisors, unit teachers, assistant 
coordinators, and regional supervisors provided additional data 
about the summer program. O.E.A. tabulated responses on the 
questionnaires and examined the comments made by program 
personnel. Observation schedules completed by O.E.A. staff 
furnished supplementary descriptive information on the 
implementation of the program. 
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III. EVALUATION FINDINGS 

PROGRAM IMPLEMENTATION 

D.S.E.'s Chapter 1 Summer Program provided reading and math 
remedial instruction a'^d athletic and recreational activities to 
Chapter 1-eligible students with mild to moderate handicapping 
conditions. Almost fifty percent of the students attended at 
least 20 sessions out of possible 30 in summer 1986. The mean 
number of sessions attended was 19.1 ( S.D> =6.6) . Sixty-four 
percent of participating students had attended at least 20 
sessions in the previous year. 

Program Start-up and Student Enrollment 

Five out of the seven site supervisors indicated that late 
notification of program funding (early June) resulted in 
difficulties in student enrollment. According to these 
supervisors, regional supervisor mailed letters to parents in mid 
or late June informing them of their child's acceptance. Some of 
these letters were not delivered because of an incorrect address, 
and others were returned after the end of the school year. At 
one site, the supervisor reported to O.E.A. that of 225 letters 
initially sent out, 25 percent were returned because of incorrect 
addresses. Data on this was not available from the previous 
cycle so it was impossible to make comparisons. To solve this 
problem, three of these site supervisors strongly recommended 
that funding notification be given early enough so that student 
acceptance and the necessary follow-up could be done while 



7 

16 



students were still in school. 



Orientation and Staff Development 

Chapter 1 provided two days of on-site orientation for all 
classroom teachers. Most of the teachers who attended the 
orientation sessions reported that these were useful in 
familiarizing them with the objectives of the program in the 
areas of testing, instructional and recreational activities, and 
use of materials. Several recommended that they be given time 
during the orientation period to set up their classrooms. Site 
supervisors also found their own orientation to be helpful, but 
three of seven interviewed suggested that it be more 
participatory, possibly using a small-group format in order to 
discuss past problems and devise solutions for the current 
program. 

With reference to staff development during the program's 
operation, seven of the 14 teachers interviewed and four of the 
seven site supervisors interviewed recommended more involvement 
of the teacher trainers on an ongoing basis. Two specific 
recommendations were that trainers offer more instruction on the 
holistic approach to reading (including demonstration lessons) 
and give guidance in working more effectively with paraprofes- 
sionals. Several staff members also recommended that one hour a 
week be devoted to staff development. 

Implementation of Assessment Procedures 

Staff at four of seven sites reported that they had received 
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sufficient information on functional reading and math levels or 
test scores for most of their students so they could select the 
correct pretest level. A number of teachers and site supervisors 
at all seven sample sites recommended that students for whom no 
pre-program functional reading and math level information was 
available be tested informally as soon as they began the program 
to determine which level M.A.T. they should administer as a 
program pretest. Program staff at six of the seven sites 
indicated that test materials had been delivered during the first 
week of the program, enabling pretesting to proceed on schedule. 

Program teachers felt that the pretest Reading and Math 
M.A.T. were useful for instructional planning for high-- and 
middle^level students, but less useful for lower-level students. 
To assess this group the teachers relied more heavily on their 
judgment and informal testing. 

Instructiona l Me tho ds and Materialjs 

Reading. In documenting reading instruction, consultants 
noted that teachers used a holistic approach in a majority of the 
classes they observed, and that teachers incorporated sports and 
Statue of Liberty themes (to coincide with its centennial 
celebration) into r.osr lessons. Teachers often included specific 
skills instruction witn holistic teaching. In one class, for 
example, the teacher asked students to define words from a "Lady 
Liberty Glossary" tnat she had written on the blackboard, use the 
words in a sentence, and discuss their meaning in relation to the 
Statue of Liberty and inirigration themes. The students read 
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aloud, then silently, a paragraph on the board containing the 
words. Students answered questions, in writing, which tested 
their comprehension and tapped specific skills such as finding 
the main idea, sequencing, and reading for detail. Finalxy, the 
teacher discussed with the students the paragraph they had read. 
In other classes, students discussed trips they had taken and 
then composed compositions, wrote in journals, or made experience 
charts describing the trips. 

Both teachers and supervisors reported that they liked using 
the holistic approach to teach reading. They felt it was 
appropriate for their students, and particularly liked the 
emphasis on language development through listening, reading, 
writing, and speaking. In addition, teachers thought that the 
stress on meaning and the integration of personal experience, 
along with the incorporation of Statue of Liberty and sports 
themes, enhanced the effectiveness of the instruction. Teachers 
reported that the staff development activities contributed to 
their successful use of the holistic approach. 

Consultants documented a variety of reading materials, 
including both commercial items and those that had been designed 
by teachers and students while teachers ana site supervisors 
were generally satisfied with the reading materials provided, 
they did note some problems and offered solutions. The most 
frequently cited difficulty was a lack of materials on the lower 
reading level as had been reported in the previous program cycle. 
To remedy this situation, three staff members recommended that 
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the program order an abundance of materials for every level and 
arrange with the publishers to return unused items. Other 
teacher recommendations were to order more audio-visual 
materials, especially tape read-alongs, more story books, and 
more of the craft supplies (e.g. construction paper, crayons, and 
scissors) that the Teacher Guide in Communication Arts 
recommended for use. 

MaUi. In math instruction, field consultants observed that 
teachers emphasized the use of word problems, whole number 
operations, practical skills, and concrete manipulative 
materials. In addition, they incorporated sports. Liberty 
themes, and field trips into the lessons. In one class, for 
example, the teacher taught the concept of "rounding off" by 
asking students to approximate and add in their heads the prices 
of tickets to a baseball game. Several teachers asked students 
to calculate the dimensions of athletic fields, keep score, trace 
baseball team standings, or compare times in races they ran. 
Teachers also reported they used the Statue of Liberty theme to 
teach measurement. Some students added the prices of the items 
they bought on a recent field trip to the zoo. Teachers used 
concrete-manipulative materials for lessons in numeration, 
addition, and subtraction. O.E.A. consultants found a 
correspondence between the math instruction they observed and the 
instructional strategies teachers reported that they used. 

Consultants observed that teachers used both commercial and 
teacher- designed math materials. Program staff reported that 
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there was an adequate supply of most materials, except for 
concrete-manipulative items. Several teachers recommended that 
the program or'^er more of these materials. 

Curriculum Guide. O.E.A. examined the Teacher's Guide in 
Communication Arts and teachers reported on its usefulness. The 
Special Education Curriculum Unit developed the Teacher's Guide 
for use during the Chapter 1 summer program. It is not used 
during the academic year. O.E.A. assessed the guide's clarity, 
comprehensiveness, and appropriateness to the goals of the 
program. With regard to clarity, the guide contained 
demonstration lessons which were well-organized and clearly 
presented. The guide gave specific directions for reinforcement 
and follow-up activities; the overall format was easy to follow. 
The guide was comprehensive in that it covered all the reading 
objectives described in the Minimum Teaching Essentials. The 
instructional objectives were considered appropriate for high- 
and middle-level readers, but judged to be too difficult for low- 
functioning readers. Finally, the guide was considered 
appropriate to the program goals for reading because lessons were 
constructed holistically ~ each lesson included extensive 
activities in reading, writing, listening, and speaking, and 
suggested methods of utilizing students' personal experiences. 
Every lesson was constructed around a Statue of Liberty or 
sports theme. 

Ten out of 14 teachers interviewed (71 percent) , indicated 
that the guide was at least moderately useful to them; they found 
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the lessons clear, well-organized, interesting, and appropriate 
with good use of thematic material. This compares with 11 out of 
13 tearhers interviewed (85 percent), last year. Eight teachers 
(57 percent) indicated that the lessons were better suited to 
high-level readers; seven teachers (50 percent) recommended that 
new guides include complete math lessons. Teachers made these 
recommendations in the last program cycle. 

Non-Instructional Activities 

Sports Activities and Tr-ipg Field consultants noted that 
the sports activities they observed were well-organized and 
included a variety of activities (e.g. basketball, swimming, 
calisthenics, dodgeball). Staff supervision, instruction, and 
student participation were judged to be satisfactory at all sites 
visited; most children appeared active and seemed to be enjoying 
the experiences. 

All staff members reported that the sports activities and 
trips were an essential part of the program. They said that 
students not only enjoyed these activities, but that their 
participation in them was motivating, bolstered their attendance 
and increased student achievement. In addition, a number of 
teachers and site supervisors expressed the opinion that the 
quality of athletic instruction provided was quite high. Most 
respondents recommended that, if possible, tne Big Apple Program 
continue over the entire si:: weeks rather than just four weeks of 
the program. The ccher recommendation, offered at three sites, 
was that there be more communication between th ^ Big Apple and 
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school staff so that teachers would be more aware of what sports 
activities were planned and the Big Apple staff could be more 
fully apprised of students' needs. 

Technical Support Services . O.E.A. evaluated three aspects 
of technical support services important to the program's 
implementation: transportation, duplication services, and 
telephones . 

Site supervisors and program teachers at five of the seven 
sample sites reported that transportation services were good. At 
the other two sites, supervisors indicated that during the first 
week of the program some buses delivered students late, 
apparently because bus drivers were not completely familiar with 
their routes. To remedy this problem, the supervisors 
recommended chat drivers become familiar with their bus routes 
prior to the start of the program. 

At three rites, both supervisors and program teachers said 
that not having a working duplicating or rexograph machine 
available on-site was a serious problem. At one of these sites, 
no telephones were available until after the program began. 

Recordk eeping . After reviewing student records at each site 
they visited, O.E.A. consultants concluded that records were 
comprehensive, well -organized, and up-to-date. 

STUDENT OUTCOMES 
Reading Achieve ment 

Each student received one of four levels of the M.A.T. 
reading test: Primary 1, Primary 2, Elementary, or Intermediate. 
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The functional reading level or earlier test score determined the 
selection of the test level. The evaluation objective in reading 
was attained in that 78.5 percent of the students across all test 
levels demonstrated a raw score gain of at least two points from 
the pretest to the posttest. (See Table 1.) Groups tested at 
all test levels met the 75 percent achievement criterion of the 
evaluation objective. (See Table 2.) The group taking the 
Elementary level had the largest proportion of students 
demonstrating a gain. The mean raw score gains at all four test 
levels were statistically significant (2 < .01). (See Table 3.) 
The mean gain ranged from a low of 5.1 (Primary 1 and 
Intermediate) to a high of 5.6 (Elementary). The mean gain 
across all test levels was 5.3 ( S.D. = 6.6). Effect size,* an 
index of the educational meaningf ulness of a gain, indicated 
that all gains were substantial, with the gain at the Elementary 
level being the most meaningful. 

Mathematics Achievement 

As in reading, each student received one of four levels of 
the M.A.T. Math test: Primary 1, Primary 2, Elementary, or 
Intermediate. Each student received only one specific math 
subtest: whole number operations, problem-solving, geometry and 



*The effect size, developed by Jacob Cohen, is a ratio of the 
mean gain to the standard deviation of the gain. This ratio 
provides an index of improvement in standard deviation units 
irrespective of the size of the sample. Effect size (E.S.) is 
interpreted to indicate educational meaningf ulness , and an E.S. 
of .8 is thought to be highly meaningful, while one of .2 is 
considered to be only slightly so. 
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TABLE 1 



Frequency Distribution of Raw Score 
Gains in Reading* 



Raw Score 


Number 
Gain 


of 


Cumulative 
Students PercentPercent 


Over 30 


9 




0.7 


0.7 


21-30 


37 




2.8 


3.5 


11-20 


148 




11.3 


14.8 


5-10 


271 




20.6 


35.4 


3-5 


415 




31.6 


67.0 


2 


151 




11.5 


78.5 


1 


124 




9.4 


87.9 


0 


50 




3.8 


91.7 


(loss) 


109 




8.3 


100.0 


TOTAL 


1, 


314 


100.0 




Source: M. 


A.T. reading 


pre- and 


posttest results. 



"The difference between pretest and posttest scores on all 
tests levels combined. 

» Over 78 percent of the participating students 

showed a raw score gain of at least two points from 
the pretest to the posttest. 

• Almost 88 percent of participating students showed a 
raw score gain of at least one point. 
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TABLE 2 



Number and Percent of Students 
Showing . Raw Score Gain of at Least Two Points 
in Reading, by Test Level 



Students Showir.g a Gain 
of at Least Two Point s 
Level Studentsa Number Percent 



Primary 1 


331 


247 


74.6 


Primary 2 


313 


246 


78.6 


Elementary 


281 


233 


82.9 


Intermediate 


239 


182 


76.2 


All levels 


1,164 


908 


78.0 



Source: M.A.T. reading pre- and posttest results. 

■Students on whom there are pretest, posttest, and test level 
data. 



^ Students tested at all test levels met the program 

objective that 75 percent would achieve a raw score gain 
of at least two points from the pretest to the posttest. 

^ Students tested at the Elementary level had the largest 
percentage showing raw score gains. 
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TABLE 3 



Statistical Significance and 
Effect Size of Mean Raw Score 
Gains in Reading, by 
Test Level 



Level 


Number of 
Students 


Mean Raw 
Score Gain 


Standard 
Deviation 


t-value 


Effect 
Size 


Primary 1 


(357)* 


5.1 


6.3 


15.2* 


.81 


Primary 2 


(322) 


5.4 


7.2 


13.4* 


.75 


Elementary 


(295) 


5.6 


6.1 


15.8* 


.92 


Intermediate 


(300) 


5.1 


6.6 


13.2* 


.77 



*Difference is statistically significant, 2 < -01- 



Source ; Reading M.A.T. pre ? .d posttest results. 



^ Mean raw scores on al"" test levels were significantly 
higher after the instructional program. 

^ The mean raw score gain and its educational effect size 
was greatest for students tested at the Elementary 
level . 



^ Raw score gains on the Primary 1 and Elementary levels 
were highly meaningful. 
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measurement, or numeration, whatever area the teacher 
proposed to cover during the summer session. 

The evaluation objective in math was attained in that 78.8 
percent of the students showed a raw score gain of at least two 
points from the pretest to the posttest. Most students gained 
from three to ten points. (See Table 4.) Students at all four 
test levels reached the achievement criterion, with the group 
taking the Elementary and Intermediate levels having the 

highest percent of students showing a gain. (See Table 5.) 
Students met the program's objective irrespective of the subtest 
which they received. {See Table 6.) 

The mean raw score gains at all four test levels and on 
three of the four math subtests were significant <.01); the 
raw score gain on the geometry and measurement subtest was not. 
(See Tables 7 and 8.) The mean gain across test levels varied 
from a low of 4.1 (Intermediate) to a high of 5.0 (Elementary), 
with an overall mean of 4.4 (S.D. = 5.1). {See Table 7.) Gains 
demonstrated by those students taking the Primary 2 level were 
the most educationally meaningful. The mean gain across subtests 
ranged from a low of 3.1 (geometry and measurement) to a high of 
4.7 (problem solving) with a mean of 4.4 (S.D. = 5.1). (See 
Table 8) . Gains de.T.oastra ted by those students taking the 
problem solving subtest were the most educationally meaningful. 

Overall, the student achievement objectives were met, even 
though the period between test administrations was brief. 
Average gains, altnough fairly small, were statistically 
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TABLE 4 





Frequency 
Raw Score Ga. 


Distribution of 
ins in Mathematics 




Gain 


Number of 
Students 


Percent 


Cumulative 
Percent 




X J 


1 . 0 


1.0 


1 1 — ort 


xxo 


8 - 3 


9.3 




29 J 


22 • 0 


31. 3 




40X 


33 . 8 


65. 1 


2 


X o «p 


1 'I 7 
X J • / 


/ O . O 


1 


123 


9.2 


88.0 


0 


77 


5.8 


93.8 


(Loss) 


82 


6.2 


100.0 


TOTAL 


1,332 


100.0 




Source: Math 


M.A.T. pre- and 


posttest results. 





Almost 79 percent of the participating students had raw 
score gains of at least two points from the pretest to 
the posttest . 

^ More than half of the students gained from three to ten 
raw score points. 
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TABLE 5 



Number and Percent of Students 
Showing a Raw Score Gain of at Least Two Points 
in Math, by Test Level 



Level 


Students^ 


Students Showing 
of at Least Two 
Number 


a Gain 
Points 
Percent 


Primary 1 


215 


163 


75.8 


Primary 2 


261 


202 


77.4 


Elementary 


360 


289 


80.3 


Intermediate 


322 


259 


80.4 


All levels 


1,158 


913 


78.8 


Source: Math 


M.A.T. pre- and 


posttest results. 





9 students on whom there are pretest, and test level data. 



Students tested at all four test levels reached the 
program objective that 75 percent would achieve a raw 
score gain of at least two points from the pretest to the 
posttest . 



ERIC 
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TABLE 6 



Number and Percent of Students 
Showing a Raw Score Gain of at Least Two Points 
in Math, by Subtest 
(N = 1,140) 





Subtest 




Students Showing a Gain 
of at Least Two Points 




Students 


Nu' *>er 


Percent 




Whole Number 
Operations 


560 


429 


76.6 




Problem 
Solving 


395 


327 


82.8 




Geometry and 
Measurement 


10 


8 


80.0 




Numeration 


175 


136 


77.7 



Source: Math M.A.T. pre- and posttest results. 



^ Students tested in problem solving had the largest 
percentage of students showing a raw score gain. 

^ Students met the program objective irrespective of which 
subtest they took. 
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TABLE 7 



Statistical Significance and Effect Size 
of Mean Raw Score Gains in 
Math, by Test Level 
(N - 1,158) 



Level 


Number of 
Students 


Mean Raw 
Score Gain 


Standard 
Deviation 


t-value 


Effect 
Size 


Primary 1 


215 


4.5 


6.8 


9.8* 


.66 


Primary 2 


261 


4.3 


4.6 


15.2* 


.93 


Elementary 


360 


5.0 


4.7 


20.7* 


1.06 


Intermediate 


322 


4.1 


4.5 


16.7* 


.91 



*Difference is statistically significant, 2 <.01. 



Source: Math M.A.T. pre- and posttest results. 

Mean raw scores on all test levels were significantly higher 
after the instructional program. 

^ The mean raw score gain was greatest for students tested at 
the Elementary level • 

^ The gain was most educationally meaningful (showed the largest 
effect size) for the Elementary group. 

^ Raw score gains on the Primary 2, Elementary, ani Intermediate 
levels had large associated effect sizes. 
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TABLE 8 



Statistical Significance and Effect Size 
of Mean Raw Score Gains in Math, 
by Subtest 
(N = 1,140) 



Level 



Number 
Students 



Whole Number 

Operations 560 

Problem Solving 395 

Geometry and 

Measurement 10 

Numeration 175 



Mean Raw 
Score Gain 



4.3 
4.7 

3.1 
4.3 



Standard 
Deviation 



4.8 
4.2 

10.0 
7.2 



*Difference is statistically significant, p <.01. 
Source: Reading M.A.T. pre- and posttest results, 



t-value 



22.9* 
24.6* 

1.2 
8.3* 



.90 
1 .12 

.31 
.86 



Mean raw score gains in all subtests except geometry and 
measurement were significantly higher after the instructional 
program . 

The mean raw score gain was greatest for students taking the 
problem-solving suDtest. 

The gain was most educationally meaningful (showed the largest 
effect size) for the problem-solving subtest. 
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significant and educationally meaningful with the exception of 
the geometry and measurement subtest in mathematics. 
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IV. CONCLUSIONS AND RECOMMENDATIONS 

The 1986 Summer Chapter 1 Program provided reading and math 
instruction to students at 13 sites from July 7th through August 
15th, 1986. Students received daily instruction in reading and 
mathematics and engaged in an athletic or recreational activity. 
The Metropolitan Achievement Test in reading and math provided 
pre-and post test data. 

The program achievement objectives were attained: over 75 
percent of students tested in reading and math demonstrated 
M.A.T. pre- to posttest raw score gains. In reading, over 78 
percent of the 1,314 participating students for whom there were 
pre- and posttest scores showed a gain of at least two points. 
This was comparable to summer 1985, when 71 percent showed a pre- 

to posttest gain of at least two points. In math, almost 79 
percent of the 1,332 students for whom there were 

pretest/posttest data showed a gain of at least two points. This 
is a higher proportion than the results of the previous program 
cycle when over 72 percent of the participating students showed 
raw score gains of at least two points. Students examined at 
each of four levels showed statistically significant pre- to 
posttest mean gains (e<.01); the effect size for all students in 
reading (.80) and math (.86) suggested that those gains were 
highly meaningful. 

Analysis of math achievement by subtest skill area revealed 
that students tested in the four subtest areas (whole number 
operations, problem-solving, geometry/measurement , and 
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numeration) attained the program objective that at least 75 
percent would demonstrate a pretest/posttest gain of at least two 
points. All except geometry and measurement were statistically 
significant (£ < .01.). Particularly noteworthy were the 
results for students instructed in problem solving. Students 

* tested on all levels of the math M.A.T. attained the program 
^ objective. 

Interviews with program st'^ff and observations of classes 
and athletic activities indicated that program services met 
student needs and that materials, with a few exceptions, 
satisfied the instructional needs of participating students. 
Although the Chapter 1 Summer Program was satisfactorily 
implemented, personnel interviewed suggested areas in which it 
might be improved. 

To ensure proper planning for the program, site supervisors 
recommended that parental notification occur much earlier in 
future programs. In this manner, parental acceptance would be 
earlier, allowing for the inclusion of functional reading and 
math levels in papers given to teachers before the start of the 
program. An improvement relative to the Summer 1985 program was 

• noted in that the pretests were delivered in a timely fashion. 
^ Program teachers, claimed that they would have appreciated 

receiving more pre-program training in using the holistic 
approach as well as additional on site training during the course 
of the program. Concerning materials both for teacher and 
student use, the teachers liked the curriculum guide. Special 
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Education Teacher's Guide in Communication Arts , and several 



recommended its expansion. Some teachers indicated that they 
wanted more equipment and materials to better meet the needs of 
their students. Teachers also felt, as they did in previous 
years, that students would benefit greatly if the Big Apple 

• sports activities were to be offered during the entire six weeks 

• of the program. 

The Chapter 1 Summer program was highly successful, taken in 
its implementation and in its outcomes. The combined effects of 
staff development activities, using the holistic approach in 
reading, integrating athletic activities, and providing out of 
class trips are the arraignment of program success. The 
conclusions lead to the following recommendations: 



* Ensure that the program begin in the most efficient 

manner possible by notifying parents of their children's 
acceptance into the program in a timely fashion, 
obtaining pre-program reading and math scores for all 
students . 

® Assist program teachers by requesting that teacher 

trainers provide more training on the holistic approach 
to reading instruction and schedule periodic on-site 
meetings; expand the Special Education Teacher's Guide in 
Communication Arts to include lessons for lower 
functioning readers and complete math lessons. 

^ Provide more audio-visual materials and equipment, math 
manipulative materials, and high-interest low reading 
level materials to sites requesting them, as well as more 
consumable instructional materials for students to take 
home . 

^ Maintain sports activities over the entire six weeks of 
the program. 
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